A framework for understanding user interaction with content-based image retrieval: model, interface and users

User interaction is essential to the communication between users and content-based image retrieval (CBIR) systems. User interaction covers three key elements: an interaction model, an interactive interface and users. The three key elements combine to enable effective interaction to happen. Many studies have investigated different aspects of user interaction. However, there is lack of research in combining all three elements in an integrated manner, especially through well-principled data analysis based on a systematic user study. In this thesis, we investigate the combination of all three elements for interactive CBIR. We first propose uInteract a framework including a novel four-factor user interaction model (FFUIM) and an interactive interface. The FFUIM aims to improve interaction and search accuracy of the relevance feedback mechanism for CBIR. The interface delivers the FFUIM visually, aiming to support users in grasping how the interaction model functions and how best to manipulate it. The framework is tested in three task-based and user-oriented comparative evaluations, which involves 12 comparative systems, 12 real life scenario tasks and 50 subjects. The quantitative data analysis shows encouraging observations on ease of use and usefulness of the proposed framework, and also reveals a large variance of the results depending on different user types. Accordingly, based on Information Foraging Theory, we further propose a user classification model along three user interaction dimensions: information goals (I), search strategies (S) and evaluation thresholds (E) of users. To our best knowledge, this is the first principled user classification model in CBIR. The model is operated and verified by a systematic qualitative data analysis based on multi linear regression on the real user interaction data from comparative user evaluations. From final quantitative and qualitative data analysis based on the ISE model, we have established what different types of users like about the framework and their preferences for interactive CBIR systems. Our findings offer useful guidelines for interactive search system design, evaluation and analysis.